Results for BarExamQA PR#240
Conversation
umarbutler
left a comment
umarbutler
left a comment
There was a problem hiding this comment.
I can confirm that I have reviewed and approve this PR on behalf of Isaacus.
There was a problem hiding this comment.
I think you've run model incorrectly. You should use model from mteb.get_model
There was a problem hiding this comment.
Hey @Samoed,
These results were generated by following the instructions for adding a dataset to MTEB: https://github.com/embeddings-benchmark/mteb/blob/main/docs/adding_a_dataset.md#submit-a-pr
The exact same code was used:
from mteb import MTEB
from sentence_transformers import SentenceTransformer
# Define the sentence-transformers model name
model_name = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
model = SentenceTransformer(model_name)
evaluation = MTEB(tasks=[YourNewTask()])There was a problem hiding this comment.
I see. I think it's a bit outdated. Thanks for pointing!
There was a problem hiding this comment.
Can you rerun this model using updated instruction embeddings-benchmark/mteb#2922?
There was a problem hiding this comment.
Yep so I reran the model using mteb.get_model and it seems to produce the right model meta data. I believe the sentence transformers version of the model is newer, leading to different meta data.
results/intfloat__multilingual-e5-small/c007d7ef6fd86656326059b28395a7a03a7c5846/BarExamQA.json
Outdated
Show resolved
Hide resolved
3 participants
Hi,
I’m submitting this pull request to push the results of
intfloat/multilingual-e5-smallandsentence-transformers/paraphrase-multilingual-MiniLM-L12-v2on BarExamQA.This PR is connected to embeddings-benchmark/mteb#2916, which adds BarExamQA to MTEB.
This pull request is being submitted courtesy of Isaacus, a legal AI research company.
Checklist
mteb/models/this can be as an API. Instruction on how to add a model can be found here